Skip to content

feat(usage): add opt-in prompt cache usage and cost buckets - #34

Merged
ALikesToCode merged 1 commit into
integrate/gateway-waves-1-2from
feat/prompt-cache-cost
Oct 9, 2026
Merged

ALikesToCode merged 1 commit into
integrate/gateway-waves-1-2from
feat/prompt-cache-cost

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Splits prompt-cache usage and cost into separate buckets behind PROMPT_CACHE_USAGE_BUCKETS_ENABLED, which defaults to false. While the flag is off, usage rows, SQL, costs and the Anthropic/OpenAI usage translations are unchanged, and migration 0016 is not needed.

  • With the flag on, usage rows record ordinary input, cache-read input, cache-write input and output separately.
    • Each bucket has its own count and microUSD cost.
    • Each row has a bucket_basis (measured, estimated or unknown) and a bucket_source.
  • OpenAI inclusive counts and Anthropic exclusive counts are normalized explicitly. A missing or ambiguous count stays null rather than becoming zero.
  • MODEL_PRICING_USD_PER_MILLION accepts cache_read and cache_write prices. PROMPT_CACHE_PRICE_METADATA_JSON can fill missing bucket prices for an explicitly priced model.
  • An unknown positive-count price leaves the total null, while known components stay visible.
  • Gateway response-cache hits cost zero.
  • SSE usage is merged across events with a 64 KiB line bound.
  • SQLite and the PostgreSQL control-plane store add the nullable columns when enabled. D1 gets them from migration 0016_usage_buckets.sql.
  • worker/usage-buckets-d1.mjs adds the Worker serializer and the extended insert for the D1 ledger.

Details are in docs/prompt-cache-cost.md.

Deployment

Apply 0016_usage_buckets.sql before enabling the flag. A push to main does not apply D1 migrations:

npx wrangler d1 migrations apply multillm-intelligence --remote

The migration only adds nullable columns to usage_events.

Integration

The Wave 3 integration PR wires the serializer into the Worker ledger's record and recent operations. Until then, leave the flag off in D1 mode.

Shared registrations

  • Added tests/test_prompt_cache_cost_worker.mjs to test:worker.
  • Added PROMPT_CACHE_USAGE_BUCKETS_ENABLED and PROMPT_CACHE_PRICE_METADATA_JSON to the Container env keys.
  • Added worker/usage-buckets-d1.mjs to the secret firewall inventory.
  • The pre-ledger schema fixtures (test_d1_schema_worker, test_control_users_worker, test_d1_integration) now skip 0016 along with 0007, because 0016 alters 0007's table.

Tests

  • tests/test_prompt_cache_cost.py and tests/test_prompt_cache_cost_worker.mjs: new.
  • Full Python suite: 3,207 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,231 passed.
  • npm run test:integration: 6 passed, run with the virtualenv python3 first on PATH (with the system python3 it fails the same way on the base branch, because Flask is missing).
  • cf:migrations:check reports 16 contiguous, additive migrations.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

This is part of Wave 3. It is based on the Wave 2 integration branch (#25), and the Wave 3 integration PR adds the README line.

Record ordinary input, cache-read, cache-write and output tokens and costs separately behind PROMPT_CACHE_USAGE_BUCKETS_ENABLED, default off, with cache_read/cache_write prices and migration 0016 adding nullable usage_events columns.
@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 17963d17-5c19-443e-8de6-5a4083ae59e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T22:18:57.412014Z 2bef0d4 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
multillm-proxy 2bef0d4 Oct 08 2026, 10:14 PM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2bef0d40eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"request_id": context.request_id,
}
if prompt_cache_cost.enabled() and context.kind in {"chat", "responses", "proxy", "embeddings"}:
breakdown = CostService.price_buckets(selected or requested, usage, requests=context.units)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve estimates when usage is absent

When buckets are enabled and a successful provider response contains no usage object, the estimated observation created above is discarded and price_buckets receives None. It therefore overwrites the legacy estimated cost with None, and _record does not perform its partial-usage fallback because usage is None; the request's reservation is then released without recording any spend. Providers that omit usage can consequently make repeated requests without consuming configured budgets, so price the request from the estimated observation as the disabled path did.

Useful? React with 👍 / 👎.

Comment on lines +180 to +181
metadata = _metadata(os.environ.get("PROMPT_CACHE_PRICE_METADATA_JSON", "").strip()) or {}
supplement = metadata.get(model, {})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Apply wildcard lookup to cache-price metadata

When an operator uses the supported wildcard pricing pattern, such as MODEL_PRICING_USD_PER_MILLION={"openai:*":...} with cache prices under the same openai:* key in metadata, _entry finds the explicit price but this exact-model lookup returns no supplement. Any positive cached-token count then leaves the total cost unknown despite the configured cache price. Normalize the metadata keys and resolve them with the same exact/provider/global wildcard precedence as the main pricing table.

Useful? React with 👍 / 👎.

Comment on lines +90 to +94
details = usage.get("prompt_tokens_details", usage.get("input_tokens_details"))
details = details if isinstance(details, dict) else {}
raw = count(typed.input_tokens)
read = count(usage.get("cache_read_input_tokens")) if anthropic else count(details.get("cached_tokens"))
write = count(usage.get("cache_creation_input_tokens")) if anthropic else count(details.get("cache_write_tokens", 0 if raw is not None or details else None))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve cache-count uncertainty through Responses translation

When protocol translation runs before accounting, responses_usage_to_chat omits prompt details for an explicitly reported cached_tokens: 0, while chat_usage_to_responses emits cached_tokens: 0 when the source omitted cache details. This parser consequently turns a measured zero into unknown in one direction and an unknown count into measured zero in the other, making the ordinary-input bucket and total cost depend on the route's translation direction rather than provider data. Preserve the null-versus-zero distinction through the Responses usage translators when buckets are enabled.

Useful? React with 👍 / 👎.

@ALikesToCode
ALikesToCode merged commit 75a8a53 into integrate/gateway-waves-1-2 Oct 9, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant